How does this CPU multibackend work? Does it check if there are *.onnx weights and if so use them if not use normal pytorch + candle?

How does this CPU multibackend work? Does it check if there are *.onnx weights and if so use them if not use normal pytorch + candle?

Yes, it tries to download the ONNX weights before, otherwise it rolls back to using safetensors, attaching here the code where the backend is initialized as a reference 👍🏻

https://github.com/huggingface/text-embeddings-inference/blob/57d8fc8128ab94fcf06b4463ba0d83a4ca25f89b/backends/src/lib.rs#L199-L295

Also @philschmid, see the logs below as reference on how do those look when running on CPU with a model from the Hub without the ONNX converted weights e.g. ibm-granite/granite-embedding-125m-english.

One minor nit within the logs is that it claims to have downloaded the onnx/model.onnx file as of the following messages (also seen in the screenshot above):

2025-01-03T09:31:22.895547Z  INFO text_embeddings_backend: backends/src/lib.rs:401: Downloading `model.onnx`
2025-01-03T09:31:22.916581Z  WARN text_embeddings_backend: backends/src/lib.rs:405: Could not download `model.onnx`: request error: HTTP status client error (404 Not Found) for url (https://huggingface.co/ibm-granite/granite-embedding-125m-english/resolve/main/model.onnx)
2025-01-03T09:31:22.916595Z  INFO text_embeddings_backend: backends/src/lib.rs:406: Downloading `onnx/model.onnx`
2025-01-03T09:31:22.936811Z  INFO text_embeddings_backend: backends/src/lib.rs:218: Model ONNX weights downloaded in 41.26313ms

But that's not true and can be missleading since it tries to initialize the ONNX backend when the file's not there cc @OlivierDehaene for reference (happy to open this or contribute to it within the TEI repository if needed!)